V 



WORLD INTELLECTUAL PROPERTY ORGANIZATION 

International Bureau 




PCT 

INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) . 



(51) International Patent Classification 0 : 
H04L 12/00 



A2 



(11) International Publication Number: WO 99/05821 

(43) International Publication Date: 4 February 1999 (04.02.99) 



(21) International Application Number: 



PCT/US98/15795 



(22) International Filing Date: 



30 March 1998 (30.03.98) 



(30) Priority Data: 
08/832,872 



4 April 1997 (04.04.97) 



US 



(71) Applicant: AVID TECHNOLOGY, INC. [US/US]; Metropoli- 

tan Technology Park, One Park West, Tewksbury, MA 
01876 (US). 

(72) Inventor: PAIGEN, David, S., W.; 444 Alvarado Street, 

Brisbane, CA 94005 (US). 

(74) Agent: MCGUINNESS, Lindsay, G.; Wolf, Greenfield & 
Sacks, P.C., 600 Atlantic Avenue, Boston, MA 02210 (US). 



(81) Designated States: AU, CA, CN, DE, GB, JP, European patent 
(AT, BE, CH, DE, DK, ES, FI, FR, GB, GR, EE, IT, LU, 
MC, NL, FT, SE). 



Without international search report and to be republished 
upon receipt of that report. 



DOCKET #^^o_u_ 
CITED BY APPLICANT 
DATE: 



(54) Title: A DIGITAL MULTIMEDIA EDITING AND DATA MANAGEMENT SYSTEM 




(57) Abstract 

A machine control subsystem is used to provide centralized control during broadcast in a news production system. A director 
workstation is coupled to a server, which is in turn coupled to any number of external devices that may be used during broadcast to air 
and display video or audio images. The director workstation uses a graphical user interface to display a list of events that are to occur 
during the production. The events include all events that are to be aired by any of the coupled devices. A director at the workstation may 
then control the production of all of the types of events from a centralized control station. To select an event or a sequence of events for 
production, the director may simply select the event from an event list, and the event will be broadcast without the director having to know 
particular characteristics of the device to display the event. The director workstation is coupled to a server. The server includes device 
manager software for interfacing with and controlling the operation of all of the externally coupled devices. By providing a centralized 
communication pathway between the director workstation and the devices, the server may forward status information about the devices to 
the director workstation to facilitate redirection of res sources by the director. 
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A DIGITAL MULTIMEDIA EDITING AND DATA MANAGEMENT SYSTEM 



Field of the Invention 

The present invention relates generally to an audio/video data management and editing 
5 system, and more particularly, to a method and apparatus for editing and managing multimedia: 
data in a multimedia production environment. 





Background of the Invention 

The process for producing broadcast news programs has undergone several changes over 
10 the last few decades. Changes in available technology, as well as increased competition brought 
about by the expansion of cable outlets and other news sources, have forced news broadcasters to 
use their resources more effectively while striving to differentiate their programming. 

In a typical news production operation, there are four major processes that occur during 
the production of a news program. These processes include video production, graphics 
15 production, text production and on-air operations. The results of these processes are rarely 
effectively combined until the actual broadcast of the news program. The video production 
— process included the generation-and editing of tape-based video for broadcast using videotape 
retrieved from a videotape archive or produced from one or more sources (i.e., cameras, either 
studio or field recorded). The text production process includes the scripting and editing of text 
20 gathered from several sources including a text archive. Similar to the tape production process 
and the text production process, the graphics production process includes the generation and 
editing of graphics data gathered from several sources. 

In order to produce the final news product for broadcast, the results of the video 
production process, the graphics production process and the text production process must be 
25 properly integrated during the on-air operations to produce the finished product. 

Existing news broadcast systems provide the capability for the integration of the video 
production process, the graphics production process and the on-air operations to provide 
complete management of audio and video elements of the news program from acquisition, 
through editing, distribution and on-air play. An integrated process in accordance with one such 
30 system includes a disk-based video production process integrated with a media production 

process and on air operations. The use of disk-based digital audio/video storage systems, digital 
networks, and digital nonlinear editing systems has allowed for successful integration of the 
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video production, graphics production and on-air operations. Several products are available from 
Avid Technology, Inc., Tewksbury, MA, for providing such an the integrated system. 

One digital multimedia newsroom production system from Avid Technology allows 
users of the system to create, browse, and catalog multimedia assets. A journalist workstation is 

5 provided that allows a user of the workstation to access multimedia assets stored in a multimedia 
archive. A user of the journalist workstation can create a story board of a multimedia product for 
broadcast. The story board may include frames of low resolution video from several diffcrcrlt 
sources, text from several sources such as news wire copy, archived text, and text created by the 
user while creating the story board, graphics, audio, still pictures, and any other form of 

10 multimedia data. 

The created story board includes a list of events, such as film clips, text, graphics, audio 
and still pictures which are to be displayed during specified portions of a broadcast. 
Traditionally, each of the events are forwarded to an associated device that is capable of 
displaying that event. Thus, the video clips are forwarded as a list of events to a video playback 
15 unit, the text is forwarded to a text display unit, the still pictures are forwarded to a still picture 
unit, and so on. Each of these units is individually controlled by a broadcast person. During 
broadcast, the selected events are integrated with live action. The integration of events is 
typically controlled by a director, with the director controlling the operations of each of the 
broadcast persons at each of the display devices. One drawback of such a production system is 
20 that the increased manpower required to control the sequence of events requires close 
synchronization by the director and introduces opportunities for error. 




Summary of the Invention 

A machine control subsystem is used to provide centralized control during broadcast in a 
25 news production system. A director workstation is coupled to a server, which is in turn coupled 
to any number of external devices that may be used during broadcast to air and display video or 
audio images. The director workstation uses a graphical user interface to display a list of events 
that are to occur during the production. The events include all events that are to be aired by any 
of the coupled devices. A director at the workstation may then control the production of all of 
30 the types of events from a centralized control station. To select an event or a sequence of events 
for production, the director may simply select the event from an event list, and the event will be 
broadcast without the director having to know particular characteristics of the device to display 
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the event. The director workstation is coupled to a server. The server includes device manager 
software for interfacing with and controlling the operation of all of the externally coupled 
devices. By providing a centralized communication pathway between the director workstation 
and the devices, the server may forward status information about the devices to the director 

5 workstation to facilitate redirection of resources by the director. 

One aspect of the invention is a news production system including a workstation coupled 
to a network for receiving a list of events to be broadcast during a newscast. At least one seiner 
is coupled to the workstation for controlling a plurality of different types of devices. The server 
also receives commands from the workstation identifying at least one event for broadcasting on 

10 one of the different types of devices. 



Brief Description of the Drawings 

Figure 1 is a block diagram of a news production system incorporating the present 
invention; and 

Figure 2 illustrates an exemplary graphical user interface for use in providing centralized 
control during broadcast in the news production system of Figure 1 . 




Description 

Figure 1 illustrates one embodiment of a digital multimedia newsroom production system 
20 10 that is shown to include three major subsystems, a video production system 12, a client 
server/core newsroom system 14 and an operations system 18. 

The components of the core newsroom system and the are interconnected using a digital 
network 20. The digital network 20 is implemented using an Ethernet network having a data rate 
greater than or equal to 100 Mb/s. The network 20 couples the three operative components 12, 
25 14 and 16 together via bridges 22 and 24. The operation of each of the subsystems 12, 14 and 16 
of the system are described in greater detail below. 
Video Production Subsystem 
— The video production system^ provides audio/video capture, media data editing, and 



management and control of high quality multimedia data suitable for broadcast. Multimedia 
30 data is defined as any form of information that can be represented in a digital form. The video 
production system includes a capture manager 24 coupled to control a news cutter/media 
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recorder 23. One example of a media recorder is the Avid Media Recorder™ available from 
Avid Technology Inc., Tewksbury, Massachusetts. 

The capture manager controls the conversion of the NTSC or PAL video into digital 
form, and forwards the converted data to an encoding station 26. At the encoding station 26, the 
5 video data is compressed in either JPEG or MPEG format the compressed audio/video is 
forwarded over network 20 via bridge 22 to a number of news servers 42a-42c in server 
subsystem 14. An asset manager 27 controls the indexing and storage of the compressed video 
data. A storage array 30 is also included for retaining archived audio/video data. Access to 
images stored in RAID array 30 is controlled via a browse server 32. 
10 A digital playback system 3 1 is coupled via a switch 38 to components in the video 

subsystem. The digital playback system 3 1 is a digital, disk-based playback system that manages 
the broadcast to air of multimedia data produced and stored within the video production system. 
The digital playback system plays materials stored either locally, at storage devices 30, or on the 
media server 42a-42c. In a preferred embodiment of the present invention, the digital playback 
15 system is implemented using an Avid AIRPLAY broadcast system available from Avid 
Technology, Inc. 

The number of graphics workstations, such as graphics workstation 28, are used for 
generating and editing graphics material for broadcast and storage in the video production 
system. In a preferred embodiment, the graphics workstation 28 is implemented using a 
20 MATADOR workstation available from Avid Technology. 

More details about the operations and features of the video subsystem 12 are described in 

i 

co-pending application entitled A Multimedia System with Improved Data Management 
Systems , filed on even date herewith, by Jason Loveman, et al. 
Server Subsystem 

25 The server subsystem 14 is shown to include a number of journalist workstations, such as 

video editing workstation 40, coupled to the network 20 via a 1 0 base T hub 37 to news servers 
42a - 42c. The number of journalist workstations used in the server subsystem is based on 
several factors including the amount of network activity generated by each user of the 
workstations and by the amount of delay each user will tolerate in accessing the system. In a 

30 preferred embodiment of the present invention, each journalist workstation 40 is implemented 
using an MPC III compliant workstation. 
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The journalist workstation provides access to multimedia data from a variety of sources 
and includes the tools (i.e., software) necessary to create a multimedia story board of a news 
story for broadcast. The multimedia data available to the journalist includes the low resolution 
MPEG video data captured by the media recorder 23. Each of the journalist workstations 

» 

5 advantageously includes a video port (not shown) for receiving video from, for example, a video 

* 

tape recorder. Each of the journalist workstations also includes a serial port (not shown) for 
controlling the video tape recorder. 1 
Each of the news servers 42a-42c provides the management and storage of multimedia 

i 

data in the newsroom environment. The news servers arc configured as distributed processors 
10 with mirrored data bases to provide maximum reliability and performance. Other centralized 
functions, such as communications functions, are managed by the news servers 42a-42c. The 
news servers may be implemented, for example, using an Avid NewsServer workstation 
available from Avid Technology, Inc. The news servers have external connections 122 for 
providing access to news wire services and to allow remote access to the news server from users 
1 5 external to the core news room system. 

The server subsystem 14 may also include an interface such as interface 45 for providing 
connections to the digital network 20 for user terminals such as user terminal 46. The user 
terminals may be one of several different terminals used in prior art systems primarily for text 
processing and communications functions. In addition, a number of device controllers (not 
20 shown) may also be coupled to the digital network 20 to provide control of several multimedia 
devices, such as teleprompters, from the journalist workstations. 

The journalist workstation 40 executes numerous functions in the news room. For 
example, the workstation 40 captures, stores and catalogs news wire text. It is used to create, 
store and catalog news story text. It is also used for capture, editing, playback, storage and 
25 cataloging of high resolution video. It is used for editing and browsing of low resolution video. 
All these functions are used by a journalist at the workstation 40 to create a rundown. The 
rundown is a list of events that are to occur during the broadcast. The rundown is forwarded 
from the server subsystem 14 over to the machine control subsystem 16. 
Machine Co ntrol Subsystem 
30 The machine control subsystem 1 6 includes three main functional components; one or 

more machine control servers 50, a technical directors workstation 52, and a serial port 
connector 54. The serial port device 54 provides connection pathways between the machine 
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control server and a number of other external devices. One serial port device for use in such a 
system is the Avanstar connector provided by.Digi International, Inc., Eden Prairie, Minnesota. 

Device manager software 51, executing at the machine control servers 50 and, optionally 

on devices coupled to the machine control servers 50, controls the transmission of data to the 

5 external devices, such as still image store 56, character generator 58, video switcher 60 and video 
playback unit 62. The still image store is for controlling the display of still images, stored in the 
new servers 42a-42c. The character generator 58 stores and displays character information, 





received over network 20, such as text for teleprompters, and text that is to be overlaid on an 
image (such as identification characters over a still image or video clip). The video switcher 60 
10 controls the display of video sequences stored in news server 42a-42c. The video playback unit 
62 cntrols the issuance of audio during production. The functions of the technical director 
workstation 52 and the machine control servers 50 are described in more detail further below. 
Technical Directors Workstation 

The Technical director workstation 52 provides intelligent, user-friendly, interactive 
15 control for newsroom production. The workstation may be, for example, an Avid Workstation 
running Windows NT. The control of components coupled to the Technical Directors 
Workstation is managed via a-graphicaluser-interface, such as that shown in Figure 2. 

The graphical user interface 70 allows the user to select events for playing on one of the 
devices such as still store 56 or character generator 58. The user may select one event, or may 
20 alternatively select a range of machine events which will then play in sequential order without 
stopping. An event list is provided at the user interface 70 for displaying a portion of the run 
down received from the server 14. The data sent from the server 14 to the machine control 
subsystem 16 is communicated using a suitable protocol. The protocol is a packet type protocol. 
The information for each event may include, for example, a short name, or 'slug' 70b identifying 
25 the subject matter of the event, a status 70c of the event, a tape ID 70d for referencing a video 
segment, a total time 70e of the event, a back time 70f, indicating the expected completion time 
of the event, a device field 70g, indicating the device on which the event is to be broadcast, a 



device status field 70h, indicating whether the device is currently in use, and a channel field 70i 
for indicating a channel on which the audio is to be broadcast. 
30 A user may select an event, or sequence of events, by depressing a mouse button over an 

event, and dragging the mouse to select a section of events for processing. Some available 
functions that can be performed on the event are outlined in the machine pull down menu 80. 
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These functions include a play, cue, stop, reload, and block play function. The play function 
causes the event to be forwarded to the device listed in field 70g of the event. The cue function 
also forwards an event to the device listed in its associated device field 70g, placing it behind a 
currently executing event. The stop function 80c stops the display of the event at the device, and 
5 the reload function 80d allows for the event to be reloaded at the device. 

The block play function 80e allows for multiple events to be sent as a 'block* over to the 
server. In one embodiment, a user selects an event, and all of the events suceeding the selected 
event, up until a break point, are forwarded as a group to the server for processing. Thus the 
block play function allows for all of the events until the next break period to be played in 
10 sequence, without intervention from the director. Thus the block play is a useful tool for 
displaying sequences of video clips for news programs that principally use pre-taped 
programming. 

The play function 80a is also used to support compound events. Compound events are a 
series of machine control events that are linked together at the journalist workstations 40 and 
15 appear as one single event that incorporates all of the linked events. Compound events differ 

from block play, since in block play there are still multiple events that are joined together to play 
- sequentially. The introduction. of compound-events allows forevents, such as a video, and a 

character generation font, that are broadcast together, to be linked together as one event. Each of 
the events in the compound event series include an optional offset from the beginning, so that 
20 they do not all start at once. 

For example, a video clip may include sound bits from Alan Green and Bob White. A 
compound event for the clip may look like: 
play the clip; 

at 1 3 seconds after the start, display "Alan Green" for 1 0 seconds; 
25 at 32 seconds after the start, display "Bob White" for 10 seconds; 

when the video clip is finished, display a still store image. 

The 'block play' menu item in the graphical user interface is provided for attaching 
events together. However most compound events are received explicitly from the new server 14 
as part of the run down. The compound event may be explicitly described, as in the above 
30 example, or described using a template of events, time offsets, and source materials. 

Thus, the compound events allow for support of common operations such as bringing up 
a title over a video clip at the same time the clip starts playing. The compound event allows 
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multiple events to be represented as a single, macro event in a rundown list. 

In addition to displaying the rundown list, the graphical user interface 70 additionally 
includes a device status window 82 for displaying the status of the attached devices. By 

< 

displaying the status of the devices, a user may readily determine whether or not that device is 
5 available, thus giving the user the flexibility to alter event/device assignments according to the 

present system status. 

A portion of one rundown list is shown in Figure 2. Although only one rundown is 
displayed at a given time, multiple rundowns may be stored at the technical director workstation. 
When a first rundown has completed, the director may select a next rundown from a list of 
10 rundowns, or alternatively rundowns may be automatically provided to the graphic user interface 
70 in sequence. 

> 

By providing control over all of the devices using a graphical user interface, control over 
the entire production is centralized at one workstation, thus requiring intervention by only one 
operator and removing the ambiguity presented by having multiple operators controlling multiple 
15 external devices. In addition, by providing control using a graphical user interface, device 

control may be automated, and the manual operator need not know the particulars of controlling 

eaeh-of the-specific external devices. — : - 

An additional benefit of providing the status of all of the events and devices couplecfrto 
the system in one graphical user interface is that it allows the director to over ride device 
20 assignments and channel assignments based on the status of the devices as viewed at the 

interface. For example, the director may override the channel assignment to maintain continuity 
if there are two back to back audio clips directed to different channels. In addition, if one of the 
devices becomes unavailable, the director may easily modify the run down to change device 
assignments of subsequent events. 
25 Although an example graphical user interface and associated functions have been 

described and illustrated in Figure 2, it should be understood that the user interface may be 
augmented to provide other types of functions that facilitate broadcast production. Thus the 
graphics user interface is meant only as an example, and not intended to be limiting, but rather 
intended to introduce a concept of centralized production control. 
30 Machine Co ntrol Server 

The machine control server 50 is the central contact point for all of the components in the 
machine control subsystem 16. The machine control server 50 also maintains a composite list of 
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events as defined by the rundown. When an event or group of events is selected by the technical 
director workstation, the event data, including device, event ID, or slug, and channel information, 
is forwarded to the machine control server 50/ 

The machine control server forwards the received events in the form of commands, based 
on the received type of event and device data, to the associated device using the device manager 
software 51. Device manager software is typically provided with an associated device, and 
includes protocol and data management routines for controlling communication with that * 





external device. Device manager softwarethaFis to be used with the MC servers should also be 
able to communicate with the server 14 according to the protocol. As a result, device status 
10 information may be made readily available to the user to facilitate device and channel 
reassignment during broadcast. 

The machine control server thus provides a centralized communication pathway for 
exchanging data between the server 14, the technical director workstation 52 and each of the 
coupled devices 56-62. Although only 8 devices are shown coupled to the machine control 
15 server 50, in one embodiment the server 50 is capable of supporting 32 devices simultaneously. 
As technology advances, it is envisioned that future embodiments will include the capability of 

supporting more than 32 devices 

As shown in Figure 1, in one embodiment the machine control subsystem may include 
dual servers 50a and 50b, configured to provide redundant support. Thus, in the event of a 
20 failure of one of the servers 50a or 50b, control is switched to the other server. This process is 
referred to as a 'warm start'. The warm start, which allows for a continuance of processing by 
the redundant server within two minutes, occurs in response to a combination of commands 
entered at the technical directors workstation 32. Serial lines to individual devices are switched 
to the redundant server using the AB switch 65. The switching operation may occur manually, 
25 or may be performed automatically via a command from the technical director workstation. 

The machine control server additionally supports the concept of virtual devices, as seen 
by the journalists. When writing a story, a journalist can specify a machine control event for a 
: named device that actually represents some combination of devices. For example, two AirPlays 
and a switch can be made to look like one dual channel Airplay to the journalist. 
30 Although the machine control subsystem has been described above with reference to an 

Avid News broadcast system, it should be understood that it may be readily adapted to support 
otheer types of news broadcast systems. For example, the machine control subsystem 16 can 
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also be used with News View, a news product for Novell® computer systems. A computer 
process would be provided for monitoring the News View database, and communicating changes 
to the machine control server 50 using the Avid Machine control protocol via network 20. 

Thus a method and apparatus for providing centralized control and communication during 
5 the production phase of a news broadcast has been described. A graphical user interface allows 
for a director, at one workstation, to control a variety of types of events that occur on multiple 
types of devices. By providing centralized control at the workstation, the director may track the 
availability of resources, and update the device selection as"fequired"dufing live production. In 
addition, the graphical user interface allows for sequences of events to be selected to simplify the 
10 production of multiple, pre-generated video clips. 

The foregoing embodiments are merely illustrative and not meant to be limiting, having 
been presented by way of example only. Numerous modifications and other embodiments are 
within the scope of one of ordinary skill in the art and are contemplated as falling within the 



scope of the invention as defined by the appended claims and equivalents thereto. 



15 
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CLAIMS 

1 . A news production system comprising: ■ 

a workstation coupled to a network for receiving a list of events to be broadcast during a 

i 

newscast; and 

at least one server, coupled to said workstation, for controlling a plurality of different 
types of devices and for receiving commands from said workstation identifying at least one event 
for broadcasting on one of said plurality of different types of devices. 



2. A news production system, comprising: 
10 a journalist workstation for providing a play list indicating a series of events to be 

displayed during a broadcast; 

a director workstation, coupled to said journalist workstation, for receiving said events, 
said workstation including an interface for allowing a user to select one of said events of said 
play list to be broadcast; and 
15 a server, coupled to said director workstation and to a plurality of different types of 

broadcast devices, for controlling the broadcast of an event received from said director 
workstation on one of said broadcast devices. 



3. The news production system according to Claim 2, wherein said interface further 
20 comprises: 

means for displaying said list of events to said user; 

means for said user to select one of said list of events from said display; and 
means for forwarding said selected event to said server. 



25 4. The news production system according to claim 2, wherein said interface is a graphical 
user interface, comprising event information and device status information, and wherein said 
selection of one of said event causes said event and at least one command associated with said 
event to be forwarded to said server. 



30 5. The news production system according to Claim 4, wherein said graphical user interface 
additionally includes a command menu, for selecting said command to be forwarded to said 
server with said event. 
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6. The news production system according to claim 4, wherein said command menu includes 
a block play command for causing multiple selected events to be forwarded together to said 
server. 

» 

5 7. The news production system according to claim 4, wherein said event is a compound 
event, comprising multiple events, and wherein said selection of said compound event results in 
said multiple events being forwarded together to said server. 

8. The news production system according to claim 4, wherein a selected device is also 
10 forwarded with said event from said workstation to said server. 




9. The news production system according to Claim 8, wherein said selected device is a 
virtual device representing more than one of said plurality of different devices, and wherein said 
selection of said virtual device results said server forwarding the event to at least one of said 

15 devices associated with said virtual device name. 

< 

10. A news production system comprising: 

a graphical user interface listing a plurality of events to be broadcast on one or more of a 

plurality of existing, said events subdivided by break points; and 
20 a block play command, at said graphical user interface, for forwarding a plurality of 

successive events between a selected event in said listing and an succeeding breakpoint to one or 
^ more broadcast devices. 
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